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Abstract — This paper proposes a computer vision based method for traffic sign detection 
recognition and tracking based on the color and shape of the road sign and its geometric 
attributes. The study of traffic sign detection has been of great interests and often addressed 
by a three-stage procedure involving detection, tracking and classification. Road safety is an 
issue of national concern and its impacts is on the economy, public health and the 
general welfare of the people. We report on-going efforts to develop an intelligent agent 
for detecting and tracking traffic signs for vision based Driver Assistance System (DAS). 
Initially paper describes general framework for traffic sign detection and their important 
subsystems. Generally, computer vision techniques consist of three important stages, color 
segmentation, sign detection and classification. We explore the important sub -systems of 
each of these stages, and identify their advantages and disadvantages. We propose a system 
based on the framework to detect circular and triangular road traffic signs making use of 
artificial intelligence techniques such as heuristics functions for detecting shapes. System is 
implemented in real time environment and tested on national highway and aims to help 
vehicle driver to have safer and enjoyable driving thereby concentrating on his actual 
workload. 

Index Terms — Computer vision, driver assistance system, object detection, openCV 

L Introduction 

In 21 st century, car is essential part of our life and is ready to change from luxury to convenience. Intelligent 
Driver Assistance System (DAS) will help vehicle drivers to react to changing road conditions, which can 
potentially improve safety. Road signs are an important road asset that is used by drivers to drive under 
safety regulations to avoid accidents and keep the order on the road. Recognition of road traffic signs 
correctly at the right time for that particular place is very important for any vehicle drivers to insure 
themselves and their passengers' safe journey. However, sometimes, due to many reasons like change of 
weather conditions or viewing angles, traffic signs are difficult to recognize. Therefore development of such 
an automatic system inside cars will certainly improve driving safety a great extent. 

With the expansion in road network, motorization and urbanization in the country, the number of road 
accidents have surged. Road traffic injuries (RTTs) and fatalities have emerged as a major public health 
concern. RTIs have become one of the leading causes of deaths, disabilities, and hospitalizations that impose 
severe socio-economic costs across the world [1]. 
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During the calendar year 2010, there were close to 5 lakh road accidents in India, which resulted in more than 
1.3 lakh deaths and inflicted injured on 5.2 lakh persons. These numbers translate into one road accident 
every minute and a road accident death every 4 minute. More than half of these victims are in active age 
group of 25-65 years. Following graph in Fig 1. gives the total number of accidents took place and people 
killed from 2001 to 2010 (provisional) [1]. 
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Figure 1. Number of accidents and persons involved from 2001 to 2010 In India 

In 1984, Japan initiated the work of Traffic sign detection. The main objective was to try to use various 
computer vision algorithms and methods for the detection of the objects in outdoor scenes. Thereafter 
researches formed various research groups to deal with problem. It is an important and widely researched 
topic in computer vision for road safety and autonomous ground vehicle. 

Generally, traffic signs are designed on left hand side of the roads in India. These signs need proper updating 
and maintenance. Road traffic sign detection and recognition system are developed as part of an Intelligent 
Transport Systems (ITS) that continuously monitors the driver, the vehicle, and the road to inform the driver 
in just in time about upcoming decisions regarding navigation and potentially risky traffic situations. 

II. Advanced Driver Assistance Systems 

Today' s automotive technologies strive for designing smarter and intelligent vehicles, aiming to minimize the 
number of accidents due to drivers' negligence or wrong-decision. These systems enhances the safety by 
informing driver of speed limits, road regulations and immediate danger like blind turn, school ahead, 
railway crossing, road work progress etc. Many systems aiming in enhancing drivers comfort and safety are 
in computer vision research. Lane departing warning system [2], [3], pedestrian detection [4]-[6], drowsiness 
detection [7], and traffic sign recognition [8]-[9], measuring traffic parameters [10], road hazard warning [1 1] 
are few of them. 

These systems are out of reach of common people owning family cars, since they are installed in few car 
manufactures like Volvo, BMW and Opel. Mostly such systems require sensors, radars, lidar or transmitters. 
Development of such systems comes with lots of problems. This includes i) poor visibility because of bad 
weather, illuminations change and resolution ii) occlusion of road signs and rotation iii) real time execution 
and speed of performance iv) variation of traffic signs in different countries. Our computer vision approach 
consists of the camera installed in small and family cars for capturing real-time video. We intend to give cost 
effective solution to the problem of road sign detection. 

A. Need of Standardization 

Aim of road signs is to facilitate the orderly and timely flow of traffic. They are the basic practices and 
procedures that road users follow; they manage interactions with other vehicles and pedestrians. In 1968, the 
Europe countries signed an international treaty, called the Vienna convention on road traffic, for the basic 
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Figure 2. Signs for indicating tunnels in different counties 
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traffic rules. A part of this treaty defined the traffic signs and signals. As a result, in Europe the traffic signs 
are well standardized, although not all countries are participants of these rules and local variations in practice 
may be found. In spite of appearances of traffic signs being strictly pre-scribed by the Vienna convention, 
there still exist variations between countries that have signed the treaty [12]. Fig 2 shows the variations of 
traffic sign across four different Asian countries related to school, kindergarten, and nursery ahead. All signs 
represent the same concept but there is little variation in their design aspect. The variations are seemingly 
irrelevant for a human, but might pose significant challenges for a computer vision algorithm [13]. Similarly, 
Fig 3 gives the different road signs, indicating tunnels in various countries. 

There is subtle difference between these icons that may not be noticed by human eyes for recognition but 
may influence the performance of detection algorithm. There is a need to standardize these icons, used for 
various informative signs in the early design process. Use of HCI concepts and human factors in designing 
these icons is inevitable. 

B. Problems in Traffic Sign Detection 

Even though there is lot of work in detection and recognition of traffic signs, there is lot of scope of 
improvements due to various problems in detections of road signs. Prominent problems faced are listed 
below: - 

• Bad lighting conditions make it harder to gather color and contour information of traffic signs, since 
illumination of signs varies continuously during day. 

• Occluded Traffic sings with other objects like trees. 

• Motion blur and car vibrations makes harder to retrieve images from the camera. 

• Damaged traffic signs 

III. Related Approcaches 

In most of the past traffic sign recognition techniques [14]— [17] the first step is to detect the location of each 
traffic sign in an image. There are three major approaches to detecting traffic signs: detection using color 
information, detection using shape information, and detection using both color and shape information. There 
is a wide range of color based detection techniques used by many researchers. In [18] Ghica described the 
technique of calculating the distance in RGB space between the pixel color and a reference color. They used 
thresholding approach to segment pixels in a digital image into object pixels and background pixels. Estevez 
and Kehtarnavaz in [9] suggested six modules in there system: color segmentation, edge localization, RGB 
differencing, edge detection, histogram extraction, and classification. There algorithm is capable of 
recognizing the Stop, Yield, and Do-Not-Enter traffic warning signs. 

Shape is an important attribute of road signs and found good attribute for sign detection. An adaptive driver 
support system named as Driver Advocate , merging various AI techniques, in particular, agents, ontology, 
production systems and machine learning technologies is discussed in [19]. Fleyah and Devami proposed a 
technique based on invoking the principal component analysis (PCA) algorithm to choose the most effective 
components of traffic sign images to classify an unknown traffic sign [20]. Andrzej, Yongmin and Liu used 
traditional three-stage framework involving detection, tracking and recognition. Detector captures instances 
of equiangular polygons that are filtered to extract the color information and establish the regions of interest. 
The tracker predicts the position and the scale of the detected sign candidate [21]. Their classifier compares a 
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discrete color image of the observed sign with the model images with respect to the class-specific sets of 
discriminative local regions. 

Detailed review of the traffic sign detection literature, detailing detection systems for traffic sign recognition 
(TSR) for driver assistance is given in [9]. They described the contributions of recent works to the various 
stages inherent in traffic sign detection: segmentation, feature extraction, and final sign detection. 



IV. General Framework for Road Sign Detection 



An intelligent agent is entity acting in a particular environment, whose actions are based on the perceptions 
received from the environment, in such a way its behavior is considered intelligent by expert. Several factors 
drove the decision to choose the agent architecture. First, the DA system must be robust to unanticipated 
input and to dynamic reconfiguration [22]. 

The first step in the detection phase as specified in Fig 4 is pre-processing, which may include several 
operations. This operation corrects an image, influenced by noise, motion blur, out-of-focus blur, distortion 
caused by low resolution. Secondly, feature images are extracted from the original image. These feature 
images contains relevant information of the original image, but in a reduced representation. Thereafter, the 
traffic signal needs separation from the background. This is achieved with simple segmentation techniques. 
After the segmentation phase follows, feature color and shape feature extraction part. In the last part of the 
detection phase, the potential traffic signs are detected from the segmented images, by using the extracted 
features of the previous phase. 
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Figure 4. General framework for road sign detection process 

After detection, we can further analyze the image with several operations and modify it or extract further 
necessary information from it. Thereafter, in the recognition phase, the detected traffic signs are classified 
into the necessary categories as shown in the Fig 5. 



A 


A A 


Steep Descent 

A 


Narrow Road Ahead Narrow Bridge 


Slippery Road 
^^^^ 


Loose Gravel Cycle Crossing 

A A 



A 



)( 



Same % 99.1324545749567 

DiffW 1.0922657608872 

Sign identification 
NARROW BRIDGE 



Figure 5. Classification of the detected sign using template matching 



V. Proposed system 

An Agent is a self-contained software element responsible for performing part of a programmatic process. 
Therefore, such agent contains some level of intelligence, performs their task independently, and may 
cooperate with other agents. A system to detect and recognize road and traffic signs should be able to work in 
real time and in all circumstances. We designed this part of the system using Microsoft visual C++ 6.0 with 
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OpenCV image development package. Heuristics used to identify the shapes that are based on shape 
identification on left side of the image frame. 

Proposed method work on similar framework, where circular and triangular signs are detected using color 
thresholding. This non-linear operation converts a gray- scale image into a binary image where the two levels 
are assigned to pixels that are below or above the specified threshold value. Detection of road traffic sign is 
performed using color and shape features. Proposed system is designed to detect the road traffic signs from 
an images or the real time video which is captured by Web camera. It consists of mainly three modules and 
each module play an important role to build complete system. 

• Module 1 : Pre-processing module include of following steps 

o Processing of Images or Real Time Video 
o Accept Frames from Video 
o Use of Gaussian Blur 

• Module 2: RGB to HSV conversion 

• Module 3: Detection of road traffic signs include following steps 

o Color Thresholding 

o Blob Detection 

o Contrast Stretching 

o Use of Median Filter 

o Scaling and Scan Boundaries 
This proposed system detects traffic signs from an images or real time video, then classify into one of the 
classes as Danger, Mandatory, Informative signs and finally recognize that sign and produce the output. 
Sequence diagram in Fig. 6 shows the sequence of activities in the process of identifying and classification of 
traffic signs. A live video stream is initially segmented into individual images from which a region of interest 
is extracted and converted to 2 Dimensional array (height * width). The default initialization of Webcam is 
800*600 pixels. 
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Figure 6. Sequence Diagram of proposed sign detection system 



VI. Result and discussion 

Video captured from camera is queried for each frame to continuously obtain any traffic sign on the road. 
Color space is changed from RGB to HSV before image filtering is performed. We constantly search for red 
color in the left-half region of video frame, since road signs are located at the left side of the road in India. 
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Fig 7 gives snapshot of the triangular and circular road sign detected. Detected signs are cropped and will be 
displayed on TFT monitor for driver assistance. Here we have shown the cropped road sign in the separate 
window on the right side to signify detected sign. The results of extracting circular signs tested on actual road 
traffic are shown in Fig 8. 




Figure 7. Triangular road sign detection 




Figure 8. Circular road sign detecti> 
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Figure 9. Detection of trangular road sign from successive frames captured from the camera mounted in vehicle 

System developed gives results for all types of circular and triangular signs as shown in Fig 7 and 8. 
Individual frames are tested for appearance of shape and color object in each frame to detect and recognize 
the sign as shown in Fig 9. 

Some of the signs but not essentially traffic signs are falsely detected by this method. User interface agent 
will help driver to get speech output of the road sign detected by the system. 



VII. Conclusions 

The traffic sign recognition is a very helpful driver assistance technique for increasing traffic and driver 
safety. The future intelligent vehicles would take some decisions about their speed, trajectory, etc. depending 
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on the signs detected. In this paper, we have presented agent based traffic sign detection process, which is 
often based on the general framework, discussed. Currently the video capture was using a digital camera and 
the result was shown by supplying video stream to the system. Proposed method gives good results for 
triangular and circular road signs detection. Much work needs to be done, before such DAS could be 
deployed in a real car assisting drivers in an actual driving environment. 

The system can be further enhanced using computer vision techniques, which would assist the driver in 
notifying the distance between the road sign and the current position of the car. This will help them to take 
appropriate decision. Even the system can be expanded to detect and recognize living objects like people 
crossing the roads, animals etc. 
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